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(54) Framework for managing cluster membership in a multiprocessor system 



(57) A shared-disk cluster system includes a cluster 
membership manager framework which coordinates the 
joining or leaving among ail nodes in a cluster including 
taking the multiple layers of involved subsystems 
through transitions. Subsystems are notified of transi- 
tions in particular order depending upon the transition, 
and all nodes' subsystems receiving a notification must 
process that notification prior to another layer of subsys- 
tems being notified. One of the subsystems registered 
for notification is an event manager in user space. The 
event manager carries out transfers of client services, 
including user applications, resulting from nodes joining 
and leaving the cluster. This Includes a registration and 
launch service which registers a node, or multiple 
nodes, in a cluster which claims, or is assigned, respon- 
sibility for the sewice and provides an optional launching 
functkin which initiates the client service upon success- 
ful registration. 
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Description 

This invention relates generally to multiprocessor systems and, more particularly, to shared-disk cluster systems. 
More particularly, the invention relates to a framework tor joining and disjoining nodes in a multiprocessor cluster 
system. 

A multiprocessor cluster system typically includes multiple nodes, which are interconnected with a private com- 
munication interconnect. The cluster system additionally includes a shared cluster resource, such as a virtual hard 
disk, which is accessible to alt of the nodes, which run an operating system supporting coordinated access to the 
shared resource. Cluster systems have many advantages. They provide high availability to the user because availability 
does not depend upon all of the nodes being active participants in the cluster. One or more nodes may leave the cluster 
without necessarily affecting availability. New nodes may be added to the system without requiring that the system be 
taken down and rebooted. Additionally, nodes may Incorporate processor designs that are different from one another, 
which facilitate expansion of the system. In this manner, the cluster system provides high aggregate performance. 

Shared-disk cluster systems have typically been used for database sen/ices which require a distributed lock system 
in order to avoid contamlnatton of data on the shared virtual disk. Membership management in such a cluster system 
required providing cluster awareness to the distributed lock system. However, such shared-disk cluster systems have 
been limited because cluster awareness extends to only one layer of subsystem. Particular operating systems have 
multiple subsystems which are layered in a manner that a higher level subsystem must depend upon the operation of 
lower level subsystems. Known cluster membership management techniques are not capable of taking such layered 
subsystems through cluster transitions of nodes joining and leaving the cluster. 

Client services are typically distributed among the nodes of the cluster requiring extensive coordination of which 
node implements which service. This is especially difficult during node transitions of a node joining or leaving the 
cluster This is because most services are not aware of the cluster environment. The client services would typically 
determine on their own the best node to execute on. A recovery mechanism would be required for initiating recovery 
If the node currently executing the service leaves the cluster. Allowing individual services to implement their own mech- 
anism for this coordination requires detailed modifications to the client services to allow them to run on a cluster system 
which makes administration of the cluster more burdensome and difficult because inconsistent mechanisms may be 
used. 

The invention in its various aspects is defined in the Independent claims below, to which reference should now be 
made. Advantageous features are set forth in the appendant claims. 

A preferred embodiment of the invention,- described in more detail below with reference to the drawings, provides 
a method and apparatus for combining particular processors, or nodes, of a multiprocessor system in a cluster that 
appears substantially as a unified processor to users of the system. Multiple subsystems running on nodes presently 
in the cluster are notified of transitions of nodes joining and leaving the cluster. This provides a consistent view of active 
membership in the cluster to the subsystems of the cluster nodes whereby all of the node's subsystems may be taken 
through the node transitions. This feature is particularly useful with subsystems that are Interdependent in levels, with 
higher level subsystems depending on the operatbn of lower level subsystems. A particular transition is noticed to the 
same level subsystem on all nodes. Notification will not proceed to another subsystem level until the noticed subsystem 
of each node processes that notification and acknowledges that such processing has been completed. When the 
transitbn is a node joining the cluster, subsystems are notified beginning with lower level subsystems and proceed in 
sequence through higher levels of subsystems. When the transition is a node gracefully leaving the cluster, subsystems 
are notified beginning with higher level subsystems and proceeding in sequence through lower level subsystems. When 
the transitbn is a node being ungracefully forced from the cluster by other nodes, subsystems are notified beginning 
with lower level subsystems and proceeding in sequence through higher level subsystems. 

A registration and launch function is provided in which client sen/ices, Including user applications, are initiated on 
particular nodes in a cluster in a manner that the cluster appears substantially as a uniform unit to the client sen/ices. 
A node is chosen for each client sen^ice and that client sen/ice is registered with the node. Nodes presently in the 
cluster are notified that the particular service is registered with the particular node. In this manner, client sen/Ices can 
be transferred to another node if the node on which that service is registered leaves the cluster. The client service may 
be launched on a node, according to an action parameter included with the service, in response to registering that 
service with that node. This provides cluster-wide availability to client services because they will not need to explicitly 
initiate themselves each time they are transferred. Client services may, advantageously, be grouped as a parent sewice 
and one or more chiW sen/ices. Grouped client sen^ices are registered with the same node and utilize action parameters 
Included with the parent service for all launching activity within the group. The choosing of a node for each client sen^ice 
may include providing a database of choosing factors for the client service and applying the choosing factors to infor- 
mation regarding the availability of the nodes In the cluster. The choosing factors establish rules relating nodes to the 
client sen/ice. 

Such registration and launch function is preferably a component of an event manager, which is a subsystem which 
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receives notification of node transitions from the cluster membership manager The event manager monitors client 
services registered with a particular node using an event watcher and provides action items which are carried out in 
response to occurrence of an event, such as a node transition. The event watcher may be enabled in response to 
registering of a client service and disabled in response to de-registering of the client service. 

The preferred embodiment of the invention will now be described in more detail, by way of example, with reference 
to the drawings, in which: 

Fig. 1 is a block diagram of a multiprocessor cluster system embodying the invention; 

Fig. 2 Is a state transition diagram of a transition notification framework for one subsystem level; 

Figs. 3-10 are diagrams of states ot subsystems in a two-node cluster illustrating nodes joining the cluster; 

Figs. 11 and 12 are diagrams of states of subsystems in a two-node cluster illustrating a graceful leave of a node 

from the cluster; 

Figs. 13-15 are diagrams of states of subsystems in a two-node cluster illustrating an ungraceful forced leave of 
a node from the cluster; 

Fig. 16 Is a block diagram illustrating the grouping of client sen^ices; 

Fig. 17 is a diagram similar to Fig. 15 illustrating multiple generations of client service groupings; 

Fig. 18 is a state transition diagram illustrating the launching of a client service; 

Fig. 19 is a state transition diagram illustrating the transition states of a client sen/ice; 

Fig. 20 is similar to Fig. 16 illustrating additional transition states; and 

Fig. 21 is a block diagram of an event manager subsystem. 

HARDWARE 

Referring now specifically to the drawings, and the illustrative embodiments depicted therein, a multiprocessor 
cluster system 25 Includes multiple nodes 26 and a shared-cluster resource, such as a physical disk 28, which could 
be made up of multiple physical disk drives (Fig. 1). Each node 26 includes a processor (CPU), physical memory, 
caches, shared and private bus interfaces, and optional dedicated devices. Each node runs a copy of a UNIX-based 
operating system, such as DG/UX 5.4 operating system marketed by Data General Corporation of Westboro. Massa- 
chusetts, running on any hardware configuration which supports such operation system. An example of such hardware 
configuration is the AViiON® family marketed by Data General. 

Cluster system 25 additionally Includes an Interconnect 36. which is a dedicated shared-cluster communication 
media that allows nodes 26 to talk directly to ail other nodes in the same cluster, and a shared-cluster I/O bus 32, which 
allows all nodes to share all devices physically connected to the shared bus, such as disk 28. In the illustrated embod- 
iment, shared bus 32 is a SCSI standard bus. 

SOFTWARE 

Cluster system 25 includes a single membership database 34, which occupies a dedicated shared-cluster virtual 
disk, which lives on physical disk 28 along with a cluster-cognizant bootstrap 38. Membership database 34 manages 
persistent node configuration informatton 40 that is needed to boot, shutdown, or panic a node 26. Such persistent 
infonmation includes Identification of the number of nodes configured with the system, as well as configuration infor- 
mation about each node, indexed by a node identification number. Membership database 34 additionally includes an 
active membership state database 42, which contains transient information about node states. Such transient infor- 
mation changes dynamically as nodes join the cluster gracefully, gracefully leave the cluster, or are ungracefully forced 
out of the cluster. A node can have any one of the following states: 

Inactive - The node is not configured or is not an active member of the cluster 

Joining - The node is in the process of joining the cluster, which implies that the node has informed other nodes 
of its intention to join the cluster gracefully, but not all of the registered subsystems of nodes in the cluster have 
completed transitions to gracefully include the new node. 

Joined - The node has fully joined the cluster and all registered subsystems of nodes in the cluster accept the new 
node as a member of the cluster and have completed their transitions to include the new node. 
Leaving - The node is in process of leaving the cluster, which Implies that the node has informed other nodes of 
its intention to leave the cluster gracefully, but not all nodes' registered subsystems of nodes in the cluster have 
completed transitions to gracefully exclude the new node. 

Forced-Leaving - Other nodes are in the process of forcing this node out of the cluster. Other nodes may force out 
a node If that node is not functioning properly, such as failing to communicate with other nodes. After the other 
nodes have completed processing of the forced-leave, which includes running recovery procedures, the other 
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nodes mark this node as inactive. The forced-out node will panic after it has noticed that the other nodes have 
forced it out. A node panics by halting further processing In order to avoid corrupting shared cluster resources. 

A. I^EMBERSHIP MANAGER 

Cluster system 25 Includes a membership manager framework including a transition notification framework 44, 
which provides notifications to all kernel-level and user-level subsystems that must receive notifications of cluster 
transittons (Figs. 2-15). The purpose of transition notification framework 44 is to provide cluster-cognizant subsystems, 
a coherent technique for processing cluster transition information among the nodes. Cluster-cognizant subsystems 
are subsystems which are registered with a node's cluster membership manager subsystem 46. In the illustrated em- 
bodiment, each node 26 includes four kernel-level subsystems, including cluster membership manager subsystem 46, 
which collectively provide transition notification framework 44, a distributed lock manager {DIM) subsystem 48, a virtual 
disk manager (VDf^) subsystem 50, and a shared file system (SFS) subsystem 52. Each node 26 additionally Includes 
at least one user-level subsystem; namely, an event manager subsystem 54. Such subsystems 46-54 are interdepend- 
ent upon each other, in levels. In the Illustrated embodiment, membership manager 46 is the lowest level subsystem 
and event manager 54 Is the highest level subsystem. However, other higher level subsystems could be provided. A 
global transition ordering is provided for the subsystems, with lower level subsystems receiving smaller values and 
higher level subsystems receiving larger values. 

Transition notification framework 44 operates as follows. Before a node joins a cluster, interested subsystems of 
that node register their intention to receive notifications of cluster transittons. A registered subsystem must also supply 
a thread of control that blocks waiting for transition notifications from membership manager subsystem 46 of that node. 
During graceful joins and forced leaves of nodes, all nodes' membership manager subsystems 46 coordinate to notify 
the node's registered subsystems in a bottom-up fashion with respect to the global transition-ordering scheme, as will 
be illustrated in more detail below. Thus, the membership manager subsystems notify, first, all of the node's subsystems 
with the lowest order followed by the next highest order, on up to the highest order. Conversely, during graceful leaves 
of nodes, all nodes' membership manager subsystems 46 coordinate to notify the node's registered subsystems in a 
top-down fashion, notifying first all nodes' subsystems with the highest order, followed by the next highest order, down 
to the lowest order. This ordering is so that higher level subsystems' dependencies on lower level subsystems are 
satisfied. That is, a lower level subsystem first processes a node join transition so that higher level subsystems can 
be ensured that the subsystems they depend upon, namely, lower level subsystems, are aware of and have completed 
processing of the join. Conversely, a higher level subsystem must first process a graceful leave so that the lower level 
subsystems remain operational in the leaving node during the leave transition. An ungraceful leave is processed from 
the bottom-up to ensure that all error conditions are propagated upward before attempting recovery at the next highest 
level. 

Each node's membership manager subsystem will not proceed with notification to the next-in-line subsystem until 
each node's currently-in-line subsystem acknowledges its completion of processing for the transition. However, each 
node's membership manager subsystem may notify a registered subsystem to process multiple transitions for different 
nodes at the same time. Each of these transitions may be of a different type. This improves performance In situatkxis 
where many nodes are undergoing transitions contemporaneously, such as when many nodes boot after a power 
failure that has powered down the entire cluster. However, each node's membership manager will not notify subsystems 
out-of-order for a particular transitional node. As a result, multiple transitions for different nodes may be processed at 
different subsystem levels at the same time, but the cluster membership manager framework will ensure proper sub- 
system ordering for each transitional node. 

For examples of use of cluster subsystems to participate In graceful joins, graceful leaves, and ungraceful forced 
leaves utilizing transition notification framework 44, reference is made to Figs. 2-15, which Illustrate a cluster system 
having potentially a two-node cluster. The examples illustrated in Figs. 3-1 5 may be generalized to three or more nodes 
with each node transition sequencing through each subsystem, one at a time, across all nodes. Each membership 
manager would not propagate the same node transition to the next highest subsystem until all nodes at the current 
level have acknowledged their completion of transition processing for the new node. 

In the state illustrated in Fig. 3, the cluster contains no active members. The system administrator powers node 
NO and begins its boot. Node N1 is left powered down. In order to gracefully join the cluster as the first active member, 
node NO opens the cluster membership database 34 and retrieves its configuration information. Node NO initializes its 
kernel subsystems 46-52, which each register themselves tor transition notification (Fig. 4). The subsystems spawn a 
thread that makes a kernel call which blocks because no cluster transitions have occurred at this time. Node NO's INIT 
subsystem (not shown) Initiates node NO's graceful join through the highest currently registered subsystem. Member- 
ship manager subsystem 46 of node NO forms the cluster and marks node NO's active state as joining. Membership 
manager 46 of node NO notifies its DLM subsystem 48 that node NO is joining the cluster (Fig. 5). The thread of DLM 
subsystem 48 of node NO Is awakened, notices node NO's new joining state, hands the joined processing off to a 
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different DLM thread, then completes its join. Node NO's DLf^ subsystem marks node NO's slate as joined and informs 
membership manager 46 of node NO that DLf^ has completed its joined processing for node NO. The same process 
is repeated for VDM subsystem 50 and SFS subsystem 52 of node NO (Fig. 6). 

After having joined the cluster at the kernel level, node NO proceeds to user space (Figs. 7a-7c). Node NO's INIT 
subsystem (not shown) spawns event manager subsystem 54, which spawns a thread which returns immediately 
because the event manager 54 of node NO has not yet processed node NO's graceful join. After having processed 
node NO'S graceful join, node NO's event manager 54 marks node NO's state as joined and informs node NO's mem- 
bership manager 46 that it has processed node NO's graceful join. 

In Fig. 8, the administrator powers and boots node N1 which causes nodes NO and N1 to perform a graceful join 
of node N1. Node N1 opens the membership database 34, retrieves it's configuration information and initializes its 
kernel subsystems 46-52. v\/hich register for transition notifications. The membership manager of joining nodes must 
negotiate with the cluster master node in order to join the cluster. When there are multiple nodes in the cluster, one 
node becomes the master node utilizing Decker's algorithm, which is known in the art. The master node writes its 
heartbeat in a particular area of membership database 34. Joining nodes will examine such area for the heartbeat in 
order to Identify the master node. Membership manager subsystem 46 of node N1 negotiates with the membership 
manager of node NO, which must be the master node because It is the only node in the cluster, in order to join the 
cluster. The membership managers of nodes NO and N1 mark the state of node N1 as joining the cluster. The mem- 
bership managers of nodes NO and N1 notify their respective DLM subsystems 48 that node N is joining the cluster. 
Both DLM subsystems wake up from their calls to begin processing node N1 's graceful join. After both DLM subsystems 
have coordinated in processing node N1's graceful join, the DLM subsystems mark node Nl's state as joined and 
acknowledge to the membership manager. After having received both DLMs' acknowledgements, the membership 
managers of nodes NO and N1 notify the respective VDM subsystems 50 that node N1 is joining the cluster (Fig. 9). 
After both VDM subsystems have processed node Nl's join, both subsystems mark node NVs state as joined and 
acknowledge the same to their respective membership managers. 

After having joined the cluster at the kernel level, node N1 proceeds to user space with its INIT subsystem (not 
shown) spawning event manager 54. Node Nl's event manager registers itself with the membership managers. Node 
N1 's event manager spawns a thread that makes a kernel call which is returned immediately because node N1 's event 
manager must process node Nl's graceful join. Node NO's event manager wakes up to process node Nl's graceful 
join. After having coordinated to process node Nl's graceful join, both event managers 54 mark node Nl's state as 
joined and acknowledge the graceful join to their respective membership managers. Node N1 is joined as illustrated 
in Fig. 10. 

A node may initiate a graceful leave while the node is still In the joining state. However, a joining subsystem will 
not convert the joining states directly to a leaving or an inactive state. The joining subsystem must complete and 
acknowledge the joined transition. The membership manager will only reverse the joining state to the leaving state 
between notifications to registered subsystem levels. 

A processing, by transitwn notification framework 44. of a graceful leave of a node, such as what would occur 
during a shutdown of a node, is illustrated by refererice to Figs. 11 and 12. Node NO initiates its shutdown by making 
the appropriate call to Initiate a graceful leave. The membership manager subsystems of nodes NO and N 1 mark node 
NO as leaving. The membership managers of nodes NO and Nl wake up both event managers 54 with node NO's 
transition. Both event managers note node NO's state as leaving and begin their coordinated processing of node NO's 
graceful leave. As will be described in more detail below, the processing of node NO's graceful leave by both event 
managers may involve a considerable amount of application level shutdown, after which both event managers mark 
node NO as inactive and notify their respective membership managers. Node NO's membership manager automatbally 
de-registers node NO's event manager for transition notification whereby node NO's event manager will receive no 
further notificatbns. Next, the membership manager of nodes NO and Nl perform the same iteration with the SFS 
subsystems 52 of both nodes, then with both VDM subsystems 50, and then DLM subsystems 48. Finally, the mem- 
bership manager of both nodes mark node NO as inactive, which also is the end of node NO's graceful leave. Node NO 
performs kernel level shutdown processing and returns to the boot command line. 

It should be noted that a node may not initiate a graceful join while other nodes are processing the node's graceful 
leave. In practice, this situation can occur when the leaving node has died abnormally and re-boots before other nodes 
have had a chance to notice that the leaving node has died. As soon as the other nodes notice that the leaving node 
has actually died, the other nodes will force the dead node out of the cluster, aborting their graceful leave processing. 
The other nodes will subsequently accept the new node's graceful join request. 

An ungraceful, forced leave is an abnornnal situation; for example, when a node is no longer capable of commu- 
nicating with the other cluster nodes. Once the forced out node notices that the other nodes have forced that node out 
of the cluster, the forced-out node panics. Transition notification framework 44 ensures that the forced out node does 
not corrupt any shared-cluster resources. When a registered subsystem is in the middle of processing a graceful join 
or leave of the forced-out node, each node's membership manager could re-notIfy the processing subsystem to abort 
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its graceful processing for the node and begin recovering processing. 

An example of a forced leave is illustrated with respect to Figs. 13-15 which begins with node N1 being joined to 
the cluster and node NO booting and joining the cluster gracefully. If, by way of example, the membership managers 
of nodes NO and N1 have processed node NO's graceful join up through the VDM subsystems 50. but have not received 
5 an aclcnowledge from the SFS subsystems 52 of their completion of node NO's graceful join processing because the 
SFS subsystems of nodes NO and N1 cannot communicate due to an interconnect failure on node NO, node NO's 
graceful join is noted as In a hung state (Fig. 1 3). The membership manager of node N1 notices that node Nl can no 
longer communicate with node NO. Node N1 forces node NO out of the cluster by marking node NO's state as forced- 
leaving. The membership manager of node NO notices that node N1 has forced out node NO and panics immediately 
10 (Fig. 14). The membership manager of node N1 Initiates forced-leave processing starting from the lowest level sub- 
system and proceeding up to the highest registered subsystem. The DLM subsystem 48 of node Nl marks node NO 
as forced-leaving, notices the abrupt transition from joined and begins recovery processing, as Illustrated in Fig. 14. 
After the DLM subsystem of node Nl acknowledges its completion of recovery processing for the forced-leave of node 
NO by marking node NO's state as inactive, the membership manager of node Nl performs the same iteration with 
IS respect to VDM subsystem 50. 

Node NO may not re-join the cluster gracefully until all of node Nl's subsystems have completed their processing 
of node NO's forced leave. When the membership manager of node Nl has finally caught up with SFS subsystem 52 
of node Nl , this subsystem will abort Its processing of node NO's original graceful join and will perform recovery process- 
ing followed by an acknowledgement of its completion of the forced leave processing for node NO by marking node 
NO'S state as inactive. The membership manager of node Nl would normally continue iterating the forced-leave noti- 
fication through the highest registered subsystem, event manager 54, However, because SFS subsystem 52 was the 
highest subsystem to be notified of node Nl's graceful join attempt, forced-leave processing will progress only through 
the SFS subsystem. After processing the forced-leave notification through the highest appropriate subsystem, in this 
case the SFS subsystem, the membership manager of node Nl nnarks the state of node NO as Inactive (Fig. 15). 

Table 1 illustrates, for a given transition node, the types of notifications that the membership manager will send to 
a registered subsystem and the corresponding acknowledgements that the membership manager expects to receive 
of the registered subsystem after the registered subsystem has completed its processing of the transitton. Table 1 also 
lists the re-notificatbns that the membership manager may send to the registered subsystem while the registered 
subsystem is still processing the original notification for the transitional node. 

Some registered subsystems may need to perfomi a two-or-more-phase commitment operation for one of the 
particular node transitions. In order to provide such multiple phase commitment, membership manager 46 provides 
barrier synchronization. Each registered subsystem may specify a number of barriers the subsystem wants for each 
type of node transition. The membership managers then provide notifications that are barrier^ynchronized with sub- 
system levels. All nodes at a given subsystem level must acknowledge Its completion of the transition processing for 
the particular barrier before the membership manager will proceed to the next-in-order subsystem. For example. If the 
DLM subsystem asks for two "joining" barriers during a joining transltbn, all DLM subsystems must acknowledge joining 
barrier 0 before they will be notified of barrier 1 . After they acknowledge barrier 1 . the joining transition will propagate 
to the VDM subsystem, which may have a different number of barriers. Also, all subsystems at a particular level must 
register with the same number of barriers for each type of transition. 

A state-diagram for transition notification network 44 is illustrated in Fig. 2 to illustrate the types of notifications 
that an individual registered subsystem may receive for a node transition. As may be obsen^ed in Fig. 2, transitioning 
may proceed through multiple barriers for each transition type. For clarrty. only one barrier Is illustrated for forced- 
leaving. However, multiple barriers are allowed Dashed lines represent transition notifications. Solid tines represent 
acknowledgements from the individual subsystem on a single node or across all nodes. 
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MEMBERSHI P MANAGER NOTIFICATIONS OF TRANSITIONS FOR A PARTICULAR NODE TO A REGISTERED 
SUBSYSTEM AND EXPECTED ACKNOWLEDGEMENTS FROM THE REGISTERED SUBSYSTEM 



TABLE 1 





Sent Notification (from 
MM to subsystem) 


New State After All 
Nodes Have 
Acknowledged 


Allowed Re- 
Notifications (from MM 
to subsystem) 


Notes 


70 
IS 


Joining 


Joined 


Forced-Leaving 


Straightforward, but this 
subsystem must keep a 
lookout for the Forced- 
Leaving re-notification and 
abort its graceful join in a 
timely fashion. 


20 


Leaving 


Inactive 


Forced-Leaving 


Straightforward, but this 
subsystem must keep a 
lookout for the Forced- 
Leaving re-notification and 
abort its graceful leave in a 1 
timely fashion. 


25 


Forced-Leaving 


Inactive 


None 


Causes this subsystem to 
abort any graceful join or 
leave processing for the 
node. 


30 


Inactive 


Inactive 


Joining 


MM should never send 
on/yan inactive notification 
to this subsystem. f^M 
sends this state along with 
real transition notifications 
for other nodes. 


35 


Joined 


Joined 


Forced-Leaving 


MM should never send 
o;?/ya Joined notif icatbn to 
this subsystem. MM sends 
this state along with real 
transition notifications for 
other nodes. || 



B. EVENT MANAGER 



Event manager subsystem 54 is a user space subsystem which provides cluster-wide availability to client services. 
This latter function is perfonned by a registration and launch service 56 (Figs. 16-21) which Is a component of event 

45 manager subsystem 54 (Fig. 21 ). Event manager subsystem 54 includes an event manager daemon 58 having multiple 
watchers 60a-60g which monitor for particular conditions. If a watcher detects a problem, the event manager subsystem 
54 will resolve the problem via action functions 62. Registration and launch sen/ice 56 may be considered a watcher 
of event manager daemon 58, but performs additional useful functions as will be explained in more detail below. 
A client sen^ice is any computing activity, including user applications, which is performed on one node or on more than 

so one node in a cluster. One difficulty is detemiining which node or nodes should initially provide each client sen/ice. Addi- 
tionally, there must be coordination of which node implements which sen/ices during a failure scenario. If individual client 
services were to implement their own mechanisms for determining the best location to execute, and for initiating recovery 
if the node currently providing the services leaves the cluster, a heavy burden would be placed upon the administrative 
management of the cluster. Registration and launch sen^ice 56 provides cluster awareness to non-cluster-aware applica- 

55 tions by choosing which node a client sen/ice will execute on. registering the client sewice with that node, and notifying 
nodes presently in the cluster that the particular client sewice is registered with that node. Registration and launch service 
56 additionally will provide an optional launching, or execution, capability, which is invoked when the sen/ice is registered 
at a particular node. The launching capability can additionally be used to transfer a service from one node to another In a 
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controlled fashion. The launching capability, referred to as "action functions, " enhances cluster-wide availability by providing 
the ability to initiate, migrate, and terminate client services. Such transfers of the client service may result, for example, 
from nodes joining and leaving the cluster. Examples of sen^ices which may make principle use of the registration and 
launch service include, by way of example, printing subsystems, floating Internet protocol (IP), networking services, and 
license sen/er. although other services may advantageously make use of the registration and launch servce. 

1 . Registration 

Registration is the process where one node, or more than one node, in a cluster can claim, or Is (are) assigned, 
responsibility for a previously defined client service. It Is performed on a cluster-wide atomic basis. A registration indi^ 
cates a claim of responsibility that the registered node is fulfilling the obligations of the specified client service. As a 
consequence of registration, or de-registration, optbnal service start-up, notification, and shut-down commands, known 
as 'action functions", will be invoked. In this manner, registration may initiate, or launch, the client service and provide 
cluster awareness to the user sen^ice. Cluster awareness Is a result of notification, upon completion of registration, to 
other nodes in the cluster, as well as nodes subsequently joining the cluster, of the registratkjn. 

2. Choosing 

The choice of which node Is assigned a particular responsibility, as part of a registration operation, is guided by a 
set of choosing parameters, or database items. 64. These choosing parameters may include a set of database items 
which specify when, where, and under what conditions a service should be registered. However, additional criteria may 
be included in the choosing function including recent performance statistics of particular nodes. Administrator-supplied 
priority factors may be selected as follows: 

AHowabIe_Nodes - The nodes from the cluster where registration Is allowed All nodes must be potential members 
of the cluster although they need not be powered up. A single wildcard character may be utilized to designate all 
potential nodes of the cluster. 

Nod© Preferences - Node preferences result from the fact that not all nodes will support all client sen^ices equally 
well. Node preferences may be specified as an unordered list or as an ordered list. Selection among unordered 
members will be influenced by recent performance characteristcs of the cluster. Ordered lists are processed be- 
ginning with the highest rank member. 

DIsallowable.Nodes - The nodes from the cluster where registration is not allowed. Adding a node to a client 
service's disaliowable node's field does not automatically initiate a transfer of the sen/ice. 
Auto_Regi8ter - This Is used when the cluster is first powered up, wherein each user client sen/ice potentially 
needs to be registered and started. The auto-register field allows the administrator to define under what conditions 
a sen^ice should be registered. 

Placement.Pollcy - This indicates what type of registering philosophy is in place; namely, whether the client 
service is to be registered on exactly one node or is to be registered and started on every allowable_node. 

As illustrated in Fig. 18, a client service may be started from either an auto-start 66 or an external start 68. The 
registration and launch service of a decision-making node selects the best node at 64 utilizing choosing parameters as 
previously set forth. The decision-making node can be any node in the cluster. It Is determined by possession of a file 
lock and is processed using a cluster-wide semaphore. If maturity applies to the client service, the registration and launch 
service transfers to a state 70 awaiting maturity. Maturity refers to the maturity of the cluster. A cluster Is mature with 
respect to a given client sen/ice if at least one node from the altowable_node's list is up and (a) the primary node is 
available, (b) enough nodes are up, or (c) enough time has gone by. Once the cluster Is nnature (70), or if maturity does 
not apply, the registratk>n and launch service notifies (72) the selected node to start the service and the other nodes 
that the client service has been assigned to a particular node. The client sen/ice is then started or launched (74). Each 
reglstratton and launch client sen^ice has independent choosing factors, except as described below (grouping). 

3. Grouping 

Registration and launch service 56 allows client services to be associated with each other in an association known 
as a grouping 78 (Figs, 16 and 17). The grouping mechanism is a relationship between a parent client service 78 and 
one or more child client 'sen^ices 80. The purpose of this grouping arrangement is to allow the administrator to specify 
associations where specific services must be placed together The child will be placed wherever the parent is placed. 
Children services do not have any choosing factors; only the parents' choosing factors are used. A grouping 76' may 
include child services 80 that are children of another service BO', which is. in turn, the child of parent servtee 78. as 
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Illustrated in Fig. 17. all of the children (80 and 80') would be under the placement of parent sen/Ice 78. 

The registration for each child 80 is pended until its parent is successfully registered and its start command, if 
any, has successfully completed. After the parent completes, each child Is processed. Similarly, a de-registration of a 
parent implies the de-registration of the children. Children are de-registered first with stop_commands invoked as 
appropriate. When transferring a grouping 76. grouped children 80 are always slopped first and started last. 

4. Action Functions 

Registration and launch service 56 supports a variety of actions to take place as a consequence of registration 
transactions. Actions are the "launch" aspect of the registration and launch service. When combined with groupings 
76 and choosing parameters 64, actions provide that many client services can depend entirely on the registration and 
launch sen^ice for cluster-wide availability. Initiation, migration, and termination may all be carried out directly with 
registration and launch service 56. 

Registration and launch sen^lce 56 In the Illustrated embodiment includes four action functions: 

Start^Command - The database is checked to determine If such command is associated with the sen^ice upon 
successful registration of the client service. If such command is present, the client sen/ice is executed on the 
registered node. The registration is not complete until the start has completed successfully. If a start operation 
fails, an attempt is made to start the client sen^ice on another node in the allowable_node list. 
Stop.Command - The database Is checked to determine the presence of this command when a client sewfce 
Intends to terminate. The de-reglstratlon Is not complete until the stop_command terminates. 
Notif y_Command - This comnnand provides a mechanism whereby other nodes are informed that the client service 
has been assigned to a particular node. When a client service is successfully registered, the database is checked 
to determine if this command is associated with the sen/ice. If. it is, It is executed on all nodes in the allowable-node 
list except the registered node. If a node In the allowable.node's list joins the cluster after a service is registered, 
and the service has a notify^command, the command is initiated on the new node. This includes nodes which leave 
and subsequently rejoin the cluster. If there Is a start.command, the notification is pended until successful start. 
Recovery.Command - This is used when a node ungracefully leaves the cluster. For each sen/ice registered on 
the forced out node, the database is checked to determine If there is a recovery_command associated with the 
service. If there is, it is executed. The node for the recovery operation is detemiined using the choosing parameters 
64. When the recovery completes, the sen^lce is de-registered. Typically, the service will then be registered and 
started on one of the surviving nodes. 

A concept closely related to an action function is that of transfer. Transfer of a sen/ice is accomplished through a 
combination of two action functions. First, the service is located and de-registered. Second, it is registered and started. 
A sen^ice transfer may be very helpful under various circurrotances. In one circumstance, the administrator may wish 
to move a sen^ice. In another circumstance, the transfer functksn Is used to transfer all of the sen/ices for a node that 
is being gracefully shut down. In another circumstance, the sen/ice, by its nature, may be trivial to move. Because 
there is no Impact to moving it, such service may be automatically transferred when a preferred node joins the cluster 
if the service is placed on a node other than a preferred node. 

When a client service is in the process of being transferred from one node to another, a "transfer intent" flag is set 
by the transferring node. Effect of the transfer intent flag on the transitions of registration and launch sen^ice 56 may 
be seen by reference to Fig. 19. Registration and launch sen/ice 56 includes a starting state 82, a registered state 84, 
a stopping state 86, a de-reglstered state 88, and a recovering state 90. Each of the stopping, starting, and recovering 
states will be skipped if their respective command does not exist. The starting state 82 indicates that the service is in 
the process of starting. If start is successful, the sen^ice goes to a registered state 84. which indicates that the service 
claims to be operational on some node. The registered sen/ice transfers to stopping state 86 if the transfer Intent flag 
is set as part of transferring to another node in a graceful leave of the operational node. The registered service transfers 
to stopping state 86 as part of an external de-register or a transfer operation. Upon completion of the stopping command, 
the sen/ice transitions from the stopping state 86 to the de-registered state 88, indicating that the sen^lce is currently 
not registered and that no node transitions are currently underaray. A sen/ice being transferred normally proceeds 
immediately from de-registered state 88 to starting state 82. Recovering state 90 only occurs If a node ungracefully 
leaves the cluster while the service was in the registered state 84 or in the stopping state 86. A more detailed state 
transition diagram is illustrated In Fig. 20. Illustrating various intermediate states. 

Table 2 illustrates an example of state transition in a three-node cluster having nodes NO, Nl, and N2. The example 
is based upon the allowable^nodes being node NO and node N1 , with a maturity_count equal to 2 and a maturity time 
equal to 5. Auto.register is set to aulo. The example applies to a single client service. At the beginning of the example, 
all nodes are down. At time t„ node N2 boots so that the status of the node changes to an upconditton. No change 
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occurs for the service because node N2 is not in the al)owable_node's list. At time the service is awaiting cluster 
maturity and node NO boots. At time tg, the maturltyjime lapses and the service start command is executed on node 
NO. Start completes at time and the service is registered on node NO. At time tg. node Ml boots. Because the service 
Is already registered, no action occurs. However, If a "notification action" Is defined, it would be executed on node N1 
at this time. At time tg. node N2 gracefully leaves the cluster, which also has no Influence on the service. At time ty, 
node NO begins a graceful leave. This results in the service being In the stopping state with the transfer intent set. 
Stopping Is completed at time tg. The sen/ice is transferred to node N1 in the starting state at time tg. The service 
becomes registered on N1 at time tio when start completes. At time t„, node NO completes the graceful leave and 
goes down. This does not represent a change In the database because the up/down status of a node is not a state 
maintained in the database. At time \^2' node N1 begins a graceful leave which places the service in a stopping state. 
When stop completes at time t^g, the service becomes de-registered and the transfer intent flag is cleared. 
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In an example illustrated in Table 3, node NO is defined as a preferred node with the remaining nodes as allowable. 
The maturlty_count is set to 2. Auto^reglster is set to auto. Placement_poljcy is set to register_on_one. Transfer.cost 
Is free. At the beginning of the example, the three nodes. NO. N1. and N2, In the cluster are all down. At time ti, node 
NO boots and the service enters the starting state on node 0. At time tg, start completes and the sen^ice is registered 
on node NO. At time tg, node N1 boots and is notified that the service Is registered on node NO. Between times t4 and 
ty, node NO gracefully leaves the cluster. The sen/ice transfers to node N1 and Is registered there. At time Iq, node N2 
boots and is notified that the service is registered on node N1 . At time tg, node NO boots and Is notified that the service 
is registered on node 1 . At time t^o, shortly after it boots, node NO notices that free transfers are allowed and that rt is 
preferred over node N1. Node NO will automatically initiate a transfer. The service enters the stopping state 86 and 
stop completes at time t^ . At time x^2^ the service enters the starting state on node NO. Starting is complete at time X^^ 
and the sewice Is registered on node NO. Nodes N1 and N2 are notified that the sendee Is registered on node NO. 
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5. Event Manager Daemon 

Registration and launch service 56 will automatically request event manager daemon 58 to monitor for a condition 
relating to a client sen/ice. Event manager daemon 58 responds by defining an event^group. The registration and 
launch service will request the event manager daemon to enable this event_group upon registration of the service and 
disable the event_group in response to de-registratbn of the service. If event manager daemon 58 detects a problem, 
an event action 62 will be invoked to resolve the problem. No direct communication will be returned to the registration 
and launch service. 

Table 4 Illustrates the fields associated with an event monitored by event manager daemon 58. In addition to the 
event name field, there are fields for IN parameters and OUT parameters which define the event that the appropriate 
event watcher is set to detect and the OUT parameters are filled in when the event occurs. The output is made available 
to the action function 62 associated with the particular event, Event_groups are used to logically associate othenvise 
independent events in order to specify when, where, and under what conditions to enable them. 



'5 TABLE 4 





EVENT MANAGER DAEMON 


1 Field 


Description 


I Event Name 


A string that identifies an event instance; it is unique within the cluster. 


IN Parameters 


A fixed set of name-value pairs that define an event; they are used by the appropriate event 
watcher to detect the event. 


OUT Parameters 


A fixed set of name-value pairs that describe an occurrence of the event. 


Action Function 


This command line describes what happens if the event occurs; it may reference values from 
the IN and OUT parameters. 



In event manager subsystem 54, event manager daemon 58 is the center of control. All watchers 60a-60g connect 

via a communication library to the event manager daemon. One of the watchers provided in event manager subsystem 
^ 54 is membership manager watcher 60e, which receives notifications from membership manager subsystem 46 of 

node transitions in the manner previously described and provides an interface to transition notification framework 44. 

Event manager subsystem 54 provides awareness to registration and launch service 56 of such node transitions. 
One example of an applicatbn for which reglstratfon and launch servfee 56 Is especially apropos is to provide a 

floating license sen/er on cluster system 25. The licensed software could be established as a sen^ice and could be 
35 allowed to execute on a given number of nodes in the cluster The registration and launch service will run the start 

program that brings up the licensed software on one of the nodes. If that node goes down gracefully, or ungracefully, 

the registratran and launch service will transfer the licensed software to a new node, after recovery if the leave was 

ungraceful. 

Th us, it is seen that the present embodiment provides a tightly coordinated cluster membership manager framework 
which coordinates the joining or leaving among all nodes in a cluster, including taking the multiple layers of involved 
subsystems through the translttons. One of the subsystems may be in user space and carries out the transfers of client 
sen/ices, including user applications, resulting from nodes joining and leaving the cluster. Other user space applications 
may register with the membership manager transition notification framework at run time. Thus, a robust system is 
provided which enhances the high aggregate performance of the multiprocessor cluster technology 

45 The present embodiment facilitates the use of multiprocessor cluster systems with operating systems having mul- 
tiple subsystems which are layered by taking all of the involved subsystems through node transitions. It also brings 
cluster awareness to non-cluster-aware client services, which include a wide variety of connputing activities including 
user applications. This allows users to treat the cluster system as a single unit with the cluster system providing cluster- 
wide availability to the client service, including initiation of the client sen^lce on a particular node, migration of the client 

50 sen/ice between nodes, and temnination of the client sen^ice. 

Changes and modifications In the specifically described embodiments can be carried out without departing from 
the principles of the mventton. which is intended to be limited only by the scope of the appended claims. 



Claims 

1. In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple subsys- 
tems on each of said nodes, a method of combining partrcular ones of said nodes in a cluster that appears sub- 
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stantially as a unified system to users of said system, including notifying subsystems running on nodes presently 
In the cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view ot active 
membership in the cluster. 

2. The method of claim 1 wherein the subsystems are interdependent in levels, with higher level subsystems de- 
pendent on the operation of lower level subsystems. 

3. The method of claim 2 including notifying one of said subsystems on all of said nodes presently in the cluster of 
a transition, processing that notification at said one of said subsystems prior to notifying another of said subsystems 
on all of said nodes presently In the cluster of the transition. 

4. The method of claim 3 including notifying subsystems beginning with lower level subsystems and proceeding in 
sequence through higher levels of subsystems of a transition of a node joining the cluster. 

5. The method of claim 3 including notifying subsystems beginning with higher lever subsystems and proceeding in 
sequence through lower levels of subsystems of a transition of a node gracefully leaving the cluster. 

6. The method of claim 3 including notifying subsystems beginning with lower level subsystems and proceeding in 
sequence through higher levels of subsystems ot a transition of a node forced from the cluster by other processors. 

7. The method of claim 2 wherein sard subsystems include a higher level subsystem which Interacts with user pro- 
grams. 

8. The method of claim 7 wherein said higher level subsystem Includes a sen/Ice which automatically and atomically 
transfers user programs to other nodes when the node executing the user programs leaves the cluster. 

9. The method of claim 2 wherein said subsystems include a distributed lock manager subsystem, a virtual disk 
manager subsystem and a shared file subsystem. 

10. The method of claim 1 wherein for a transition of a node joining the cluster, said method includes the steps of: 

a) registering subsystems of the joining node to receive transition notifications; 

b) joining the node to the cluster; and 

c) notifying registered subsystems in the cluster that the joining node has joined the cluster 

11. The method ot claim 1 wherein for a transition of one node being forced out of the cluster by another node, said 
method includes the steps of: 

a) the another node notifying registered subsystems that the one node is being forced out of the cluster; and 

b) transferring registered programs executing on said one node atomically to a different node and recovering 
the programs to execute on said different node. 

12. In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple subsys- 
tems on each of said nodes, a method of initiating client sen/ices on particular ones of said nodes in a cluster in 
a manner that appears substantially as a unified system to the client services, including choosing a node for each 
client sen/ice, registering the client sen/ice with that node, and notifying nodes presently in the cluster that the 
particular client sen^ice Is registered with the particular node, whereby the particular service can be transferred to 
another node If the particular node leaves the cluster. 

13. The method of claim 12 further including launching a client service on a node according to an action parameter 
included with the client service In response to registering that client sen/ice with that node. 

14. The method of claim 1 3 further including grouping client services as a parent client service and at least one child 
client semce, registering grouped client services with the same node and launching grouped client sen^ices ac- 
cording to an action parameter included with the parent client sen/ice. 

15. The method of claim 12 wherein said choosing a node Includes providing a database ol choosing factors for the 
client sen^ice and applying said choosing factors to the nodes presently in the cluster, said choosing factors es- 
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tabiishing rules relating nodes to the client sen^ice. 

16. The method of claim 15, wherein said choosing factors are selected from the group Including allowable nodes, 
disallowable nodes and node preferences. 

17. The method of claim 12 further including notifying nodes joining the cluster that the particular client service is 
registered with the particular nodes. 

18. The method of claim 12 including monitoring a client service registered with a node at that node using an event 
watcher. 

19. The method of claim 18 including enabling the event watcher in response to registering the client service and 
disabling the event watcher in response to de-registering the client service. 

20. The method of claim 1 2 wherein said multiple subsystems includes a cluster membership manager which controls 
which of said nodes are presently In the cluster and wherein said cluster membership manager provides notification 
of which nodes are in the cluster. 

21. The method of claim 20 wherein said Initiating sen/ices on a particular one of said nodes is performed by another 

one of said multiple subsystems. 

22. A multiprocessor cluster system having multiple nodes, a shared resource accessible to all nodes, a cluster com- 
munication medium between said nodes, and multiple subsystems on each of said nodes, comprising: 

a cluster membership manager subsystem adapted to notify subsystems running on nodes presently in the 
cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active 
membership in the cluster; 

an event manager subsystem adapted to detect and react to cluster errors; and 

a registration and launch service responsive to said event manager and adapted to initiate client services on 
particular ones of said nodes in a cluster in a manner that appears substantially as a unified node to the client 
services, wherein said registration and launch subsystem chooses a node for each client service, registers 
the client sen^ice with that node, and notifies nodes presently in the cluster that the particular service is reg- 
istered with the particular node. 

23. The multiprocessor cluster system in claim 22 wherein the subsystems are interdependent in levels, with higher 
level subsystems dependent on the operation of lower level subsystems, 

24. The multiprocessor cluster system in claim 23 wherein said membership manager subsystem notifies one of said 
subsystems on all of said nodes presently in the cluster of a transition, and that one of said subsystems on all 
nodes processes that notification prior to said membership manager subsystem notifying another of said subsys- 
tems on all of said nodes presently in the cluster of the transition. 

25. The multiprocessor cluster system in claim 24 wherein said membership manager notifies subsystems beginning 
with lower level subsystems and proceeding in sequence through higher levels of subsystems of a transition of a 
node joining the cluster 

26. The multiprocessor cluster system in claim 24 wherein said membership nnanager notifies subsystems beginning 
with higher lever subsystems and proceeding in sequence through lower levels of subsystems of a transition of a 
node gracefully leaving the cluster. 

27. The multiprocessor cluster system in claim 24 wherein said membership manager notifies subsystems beginning 
with lower level subsystems and proceeding in sequence through higher levels of subsystems of a transition of a 
node forced from the cluster by other processors. 

28. The multiprocessor cluster system in claim 22 wherein said registration and launch sen/ice launches a client service 
on a node according to an action parameter included with the client service in response to registering that client 
service with that node. 
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29. The multiprocessor cluster system In claim 26 wherein said registration and launch sen/ice is adapted to group 
client services as a parent client service and at least one child client sen^ice, and wherein said registration and 
launch service further registers grouped client services with the same node and launches grouped client services 
according to an action parameter included with the parent client service. 

30. The multiprocessor cluster system in claim 22 wherein said registration and launch service includes a database 
of choosing factors for the client service and applies said choosing factors to the nodes presently in the cluster to 
choose the node for registering a client service, said choosing factors establishing rules relating nodes to the client 
sen^ice. 

31. The multiprocessor cluster system in claim 22 wherein said registration and launch service further notifies nodes 
joining the cluster that the particular client service is registered with the particular node. 

32. The multiprocessor cluster system In claim 22 wherein said event manager includes an event watcher for nnonitorlng 
a client sen/ice registered with a node at that node. 

33. A computer usable medium in which program code is embodied, said program code defining an operating system 
for a multiprocessor cluster system having multiple nodes, a shared resource accessible to all processors, and 
including multiple subsystems, one of said subsystems being a cluster membership manager subsystem adapted 
to notify subsystems running on nodes presently in a cluster of transitions of nodes joining and leaving the cluster 
in order to provide a consistent view of active membership in the cluster. 

34. A computer usable medium in which program code Is embodied, said program code defining an operating system 
for a multiprocessor cluster system having multiple nodes, a shared resource accessible to all processors, and 
including a registration and launch sen/Ice adapted to initiate client sen/ices on particular ones of nodes in a cluster 
in a manner that appears substantially as a unified system to the client services, wherein said registration and 
launch service chooses a node for each client service, registers the client sen^ice with that node, and notifies nodes 
presently in the cluster that the particular service is registered with the particular node. 

35. A computer usable medium in which program code is embodied, said program code defining an operating system 
for a multiprocessor cluster system having multiple nodes, and a shared resource accessible to all nodes, com- 
prising: 

multiple subsystems that are interdependent in levels, with higher level subsystems dependent on the oper- 
ation of lower level subsystems; 

one of said subsystems comprising a cluster membership manager subsystem adapted to notifying subsys- 
tems running on nodes presently in a cluster of transitions of processors joining and leaving the cluster in 
order to provide a consistent view of active membership in the cluster; and 

one of said subsystems including a registration and launch sewice adapted to initiate client services on par- 
ticular ones of said nodes in a cluster in a manner that appears substantially as a unified node to the client 
sen^ices, wherein said registration and launch sen/ice chooses a node for each client service, registers the 
client service with that node, and notifies nodes presently in the cluster that the particular sen^ice is registered 
with the particular node. 

36. In a multiprocessor system having multiple nodes, and a shared resource accessible to all nodes, a method of 
initiating client sen^ices on particular ones of said nodes in a cluster in a manner that appears substantially as a 
unified system to the client sen/ice including registering a client servfce with one of said nodes and launching the 
client sen/ice on that node according to an action parameter included with the client service in response to regis- 
tering that user service with that node. 

37. The method in claim 36 including transferring the client sen/ice to another node if that node leaves the cluster. 

38. The method in claim 36 wherein said transferring includes relaunching the client sen/ice on the another node 
according to said action parameter. 



16 



EP0750 256 A2 




LL 



17 



EP0 750 256 A2 



All Nodes 
ACKed 




Joining 

noUficallon '^^ ^ Inacllve 



Fig. 2 



18 



EP0750256 A2 



54- 



52- 



50- 



48- 



46' 



E\ferM Manager 

not registered 



26 



26 



SFS 

not registered 



VDM 

not registered 



DLM 

not registered 




-44 



/ 

Event Manager 

not registered 



SFS 

not registered 



VDM 

not registered 



DLM 

nor registered 



MM [jnfI 

not registered 



54 



•52 



■50 



-48 

-44 
H-46 



Node NO Fig. 3 NodeNI 



(Inactive - booting) 



(Inactive - powered down) 



26 



54- 



52- 



50- 



Event Manager 

not registered 





46" 



] MM 'j^-^' 

'{NO:lnactlve. Nl:lnactlve) 



-44 



26 



Event Manager 

not registered 



Node NO Fiq. 4 

(Inactive - registering) ^ 



- -54 



SFS 

not registered 



VDM 

not registered 



^-50 



DLM 

not registered 



MM [jNf ^ 

not registered 



.52 



-48 

-44 
-46 



Node N1 

(Inactive « powered down) 



19 



EP 0 750 256 A2 



26 



L 

Event Manager 

not registered 



26 



SFS 

(NO:lnactlve, N1 .inactive) 



VDM 

{NO:lnacltve, Ntilnacllve) 




44 



Event Manager 

not registered 



SFS 

not registered 



VDM 

not registered 



DLM 

not registered 



not registered 



-54 
-52 
-50 

-48 

-44 
-46 



Node NO p:^ n Node N1 

(Joining) ' ^ (Inacliva - powered clown) 



26 



Event Manager 

not reglalered 





^44 



26 



Event Manager 

not registered 



SFS 

not registered 



VDM 

170/ registered 



DLM 

not registered 



MM Ijtnf' 

no/ registered 



•54 



-52 



-50 



-48 

-44 
46 



Node NO pi^ f- NodeNI 

(Joined) riy. U (inacllve - powered down) 



20 



EP0 750 256 A2 



26 

/ 



CFS 

(NO:Jolned, N1 inactive} 



VDM 

{NO:Joined, N1:lnactlve} 



DLM 

(NO:Jolned, N1 inactive) 



,(N0:Jolned.N1 '.Inactive] 



.44 



26 



Event Manager 

not registered 



CFS 

not registered 



VDM 

not reglslend 



DLM 

not registered 



not registered 



MM : TNF 



•Node NO ^. NodeNI 

(Joined) P IQ . r 3 (Inactive - powered down) 



.26 



.26 




CFS 

(NO: Joined, N1 inactive} 



VDM 

(NO:Joined, N1:lnactlVe) 



DLM 

{NO:Jolned. N1:lnactlve] 



MM[jNF^ 
(NO: Joined, N1:lnacllve) 



-44 



Event Manager 

not registered 



(Joining) 



Fig. 7b 



CFS 
not registered 



VDM 

not registered 



DLM 

not registered 



MM'-w"^ 

not registered 



Node NO r-:^ -7u NodeNI 



(Inactive - powered down) 



21 



EP0750256 A2 




26 



SFS 

(N0:Joinect,N1:lnacllva) 



VDM 

{NO:Jolned,N1:inaetlve) 



DLM 
{NO Joined, N1:lnBctlve] 



MM [jnf;3 

(NO:Jained, N1:tnact)ve) 



- 44 



26 



Event Manager 

not registered 



SFS 

nol reglstBrdd 



VDM 

nor registered 



DLM 

not registered 



MM [thf~ 

not raglslared 



Node NO Node N1 

(Joined) -y^ (Inactive - powered down) 



Fig. 7c 



26 



Event Manager 

(NO:Jolned, Nl;lnactlve) 



SFS 

{NOMolned. N1 inactive) 



VDM 

(NO.'Joinad, N1 .Inactive} 



DLM 

(NO:Jolned, N1 ilnacUve) 




- 44 



Node NO 

(Joined) 



Fig. 8 



26 



Event Manager 

not regfsiarod 



SFS 

(NOilnactlve, N1 .-Inactive) 



VDM 

(NO:lnacllva, NlilnactlvoJ 



DLM 

(N0:!nactlv6, NltlnactlveJ I 




Node N1 

(Joining) 



22 



EP 0 750 256 A2 



26 



54- 



52- 



48- 



46- 



C 

Event Manager 

(NO:Jolned, N1 inactive | 



SFS 

{NO: Joined. N1:lnBCtive) 




DLM 

{NQ:Joined, N1: Joined) 



(NO:Jolned. N1: Joining) 



-44 



26 



Event Manager 

nci registered 



SFS 

(NO:inactlve. N1:lnactlve} 



DLM 

(NOrJolned, N1:Jolned) 



(NOrJolned. N1 Joining) 



'54 



-52 



-50 



-48 

-44 
--46 



Node NO 

(Joined) 



Fin Q NodeNI 

' 'y- (Joining) 



26 




52- - 



SFS 

(NO:Joined. N1 Joined] 



50- 



48 



VDM 

{NO: Joined, N1 Joined) 



DLM 

(NOiJolned, N1:Jotned) 



46- 



MM [^TNF^ 
(NO: Joined. N1: Joined) 



44 



26 




SFS 

(NOrJolned, NIrJolned) 



VDM 

(NOiJolned. N1 Joined) 



DLM 

{NO:Jolned. Nl:Joined) 



MM [JNF^ 
{NO:Jolned, N1:Joined) 



'52 

•50 

-48 

-44 
• 46 



Node NO Fia 10 NodeNI 

(Joined) * (Joined) 



23 



EP 0 750 256 A2 



.26 



54- 




52-- 



SFS 

{NO:Jolned, NliJolned) 



50' 



48' 



VDM 

{NO: Joined, N1 Joined) 



DLM 

(NO: Joined, NliJolned) 



46^ 




-44 



26 




SFS 

{NOJolned. N1 Joined) 



VDM 

{NO: Joined. Nl:Jolned} 



DLM 
{NO:Jolned, N1:Jo]ned) 



-52 



-50 



-48 




Node NO r:\n 11 Node N1 

(Leaving) Tiy. 11 (Joined) 



26 



54' 



52- 



50- 



48- 



Event Manager 

not registered 



SFS 

not registered 



VDM 

not registered 



DLM 

pot registered 




44 



26 



Event Manager 

(NOrlnactlve, N1: Joined) 



SFS 

{NO:lnactlve, N1:Joined) 



VDM 

(NO:lnactlve, HIrJotned) 



DLM 

{NO:lnactlve, NliJolned] 



.54 



—52 



-50 



-48 




Node NO f in 10 

(Inactive) ' 



Node N1 

(Joined) 



24 



EP0750 256A2 



54- 



Event Manager 

not raglstered 




50- 



48^ 



46 



VDM 

(N0:Joln6d, N1 Joined) 



DLM 

(NO:Joined. N1:Jo)ned) 



MM 

(NOJolnlng, N1 rJolned) 



■44 



26 



Event Manager 

(NO:lnactlve, tri tJolned) 




VDM 

{NO:Jo)nad, N1:Joined] 



DLM 

{NO: Joined, N1: Joined) 



MM {jnf;5 

(NO: Joining. N1:Jolned) 



^ Node NO pia 13 NodeNI 

(Joining, but unreachable) * 'a* * (Joined) 



.26 



54- 



Event Manager 

not registered ' 



52- 



50' 



48- 



46" 



SFS 

not registered 



VDM 

not registered 



DLM 

not registered 



MM [jNF_^ 
not registered 



-44 



26 



Event Manager 

{NO:lnaetlve, N1 .Joined} 



SFS (hung) 

(NO:Jolnfng. N1:Jolned} 



VDM 

(NO: Joined, .N1 :Jolned) 




. MM {^TNF^ 
(NOrF-LeavIng, NVJolnedj 



•5 



■4 



Node NO pi^ A A NpdeNI 

(Inactive " panicked) ' 'y- (Joined) 



(Joined) 



25 



EP07S0 256 A2 



54- 



52- 



50- 



48- 



46- 



Event Manager 

not registered 



SFS 

not raglsterad 



VDM 

not registered 



DLM 

hot registered 



MM 

not registered 



TNF , 



44 



Node NO Pin -i c Node ^ 

(Inacllve - panicked) • 'y • ' (Joined) 



Event Manager 

(NO.-lnactive. Nl Joined) 



SFS 

(NO:lnactlve, NIrJolned) 



- -54 



• 52 



VDM 

{NO:lnactive. N1: Joined] ' 



DLM 

{NO:lnactlve, N1 Joined) 



■50 



-48 




Node N1 



26 



EP0 750256 A2 




27 



EP 0 750 256 A2 




ri 



C 



Uiigniccriil Lciiv( 
(xfur iiiieni scO ,|i'' 



Registered 



C Recovering y 
,^ — A 



^90 f^ccovcry CcMiiplcic 
— (xfer iiiiem injiiii mined) 



\|Ungniccfiil Lcjivc 











Suiri Succeeds 









Gxicrti.-il Dcrcgijiicr 



^84 



Key: 

1^ xl'cr imciu .set 

^ tJonnal |)ii)cc/>.siiiy 

uiiBraLciiil cxii inmi diiMcr 




Ail Slam Pail 



Fig. 19 



28 



EPO 750256 A2 




29 



EP0 750 256 A2 



TRANSITION 
NOTIFICATION 

FRAMEWORK 



^44 




Directives 



Fig. 21 



30 



